Endolumenal object sizing

ABSTRACT

An object sizing system sizes an object positioned within a patient. The object sizing system identifies a presence of the object. The object sizing system navigates an elongate body of an instrument to a position proximal to the object within the patient. An imaging sensor coupled to the elongate body captures one or more sequential images of the object. The instrument may be further moved around within the patient to capture additional images at different positions/orientations relative to the object. The object sizing system also acquires robot data and/or EM data associated with the positions and orientations of the elongate body. The object sizing system analyzes the captured images based on the acquired robot data to estimate a size of the object.

INCORPORATION BY REFERENCE TO ANY PRIORITY APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/392,674 filed Dec. 28, 2016, which is incorporated herein byreference. Any and all applications for which a foreign or domesticpriority claim is identified in the Application Data Sheet as filed withthe present application are hereby incorporated by reference under 37CFR 1.57.

BACKGROUND 1. Field of Art

This description generally relates to surgical robotics, andparticularly to sizing an object within a lumen of a patient's body.

2. Description of the Related Art

In ureteroscopy, endoscopic removal of objects (e.g., kidney stones)within a patient's body often involves use of basket to capture of thekidney stone, retraction of the basket, and retraction of theureteroscope down the ureter, bladder and out the urethra. Duringretraction of the basket and the ureteroscope, there is a risk that thekidney stone, being too large in diameter, may cause avulsion of theureter. In conventional surgical systems, practitioners like physicianstypically position an endolumenal safety wire to trace the path of theureter from the bladder to the kidney in order to guide the reattachmentof the ureter if the ureter is avulsed. The used of a safety wire,however, creates an additional step to the physician's workflow. Inaddition, kidney stones are not easily seen and measured using currentexternal viewing techniques like fluoroscopy.

Thus, there is a need for a method of endolumenally measuring the sizeof a kidney stone in order to determine whether it is safe to retractthe ureteroscope and endoscopically extract the kidney stone.

SUMMARY

The methods and systems disclosed herein allow for sizing an objectpositioned within a patient's body, for example, a kidney stone within apatient's ureter. As one example, an object sizing system sizes anobject positioned within a patient. The object sizing system identifiesa presence of the object. The object sizing system navigates an elongatebody of an instrument to a position proximal to the object within thepatient. An imaging sensor coupled to the elongate body captures one ormore sequential images of the object. The instrument may be furthermoved around within the patient to capture additional images atdifferent positions/orientations relative to the object. The objectsizing system also acquires robot data and/or EM data associated withthe positions and orientations of the elongate body. The object sizingsystem analyzes the captured images based on the acquired robot data toestimate a size of the object.

A variety of specific techniques (and associated system components) maybe used to estimate the size of the object,. As one example, the objectsizing system generates a stitched image from multiple captured imagesto identify a structure of the object. As another example, a roboticbasket may be used to capture the object for determining a size of theobject. As a further example, structured light can be used to illuminatethe object, and the object sizing system estimates a size of the objectbased on illumination patterns projected on the object.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A shows an example surgical robotic system, according to oneembodiment.

FIGS. 1B-1F show various perspective views of a robotic platform coupledto the surgical robotic system shown in FIG. 1A, according to oneembodiment.

FIG. 2 shows an example command console for the example surgical roboticsystem, according to one embodiment.

FIG. 3A shows an isometric view of an example independent drivemechanism of the instrument device manipulator (IDM) shown in FIG. 1A,according to one embodiment.

FIG. 3B shows a conceptual diagram that shows how forces may be measuredby a strain gauge of the independent drive mechanism shown in FIG. 3A,according to one embodiment.

FIG. 4A shows a top view of an example endoscope, according to oneembodiment.

FIG. 4B shows an example endoscope tip of the endoscope shown in FIG.4A, according to one embodiment.

FIG. 5A shows an example schematic setup of an EM tracking systemincluded in a surgical robotic system, according to one embodiment.

FIGS. 5B-5E show example graphs illustrating on-the-fly registration ofan EM system to a 3D model of a path through a tubular network,according to one embodiment.

FIG. 6A shows a perspective view of a surgical robotics system withcolumn-mounted arms configured to access the lower body area of apatient according to one embodiment.

FIG. 6B shows a top view of the surgical robotics system withcolumn-mounted arms configured to access the lower body area of thepatient according to one embodiment.

FIG. 7A shows a side view of a basket apparatus of a surgical roboticsystem, according to one embodiment.

FIGS. 7B and 7C show how the basket apparatus may be used to capture akidney stone, according to one embodiment.

FIG. 8 shows an example block diagram of an object sizing system,according to one embodiment.

FIG. 9 shows an example flowchart illustrating a process of estimatingsize of an object within a lumen of a patient's body by the objectsizing system shown in FIG. 8, according to one embodiment.

FIG. 10 shows an example flowchart illustrating a process of estimatingsize of the object using image stitching techniques, according to oneembodiment.

FIG. 11A shows an example flowchart illustrating a process of estimatingsize of an object using basket marker detection, according to oneembodiment.

FIG. 11B shows an example flowchart illustrating a process of estimatingsize of an object using motion detection with a basket capturing theobject, according to one embodiment.

FIG. 12 shows an example flowchart illustrating a process of estimatingsize of an object illuminated by structured light within a lumen of apatient's body, according to one embodiment.

FIG. 13 shows an example flowchart illustrating a process of estimatingsize of an object within a lumen of a patient's body employing motiondetection using robot data or optical flow techniques, according to oneembodiment.

Reference will now be made in detail to several embodiments, examples ofwhich are illustrated in the accompanying figures. It is noted thatwherever practicable similar or like reference numbers may be used inthe figures and may indicate similar or like functionality. The figuresdepict embodiments of the described system (or method) for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles described herein.

DETAILED DESCRIPTION OF DRAWINGS

I. Surgical Robotic System

FIG. 1A shows an example surgical robotic system 100, according to oneembodiment. The surgical robotic system 100 includes a base 101 coupledto one or more robotic arms, e.g., robotic arm 102. The base 101 iscommunicatively coupled to a command console, which is further describedwith reference to FIG. 2 in Section II. COMMAND CONSOLE. The base 101can be positioned such that the robotic arm 102 has access to perform asurgical procedure on a patient, while a user such as a physician maycontrol the surgical robotic system 100 from the comfort of the commandconsole. In some embodiments, the base 101 may be coupled to a surgicaloperating table or bed for supporting the patient. Though not shown inFIG. 1 for purposes of clarity, the base 101 may include subsystems suchas control electronics, pneumatics, power sources, optical sources, andthe like. The robotic arm 102 includes multiple arm segments 110 coupledat joints 111, which provides the robotic arm 102 multiple degrees offreedom, e.g., seven degrees of freedom corresponding to seven armsegments. The base 101 may contain a source of power 112, pneumaticpressure 113, and control and sensor electronics 114—includingcomponents such as a central processing unit, data bus, controlcircuitry, and memory—and related actuators such as motors to move therobotic arm 102. The electronics 114 in the base 101 may also processand transmit control signals communicated from the command console.

In some embodiments, the base 101 includes wheels 115 to transport thesurgical robotic system 100. Mobility of the surgical robotic system 100helps accommodate space constraints in a surgical operating room as wellas facilitate appropriate positioning and movement of surgicalequipment. Further, the mobility allows the robotic arms 102 to beconfigured such that the robotic arms 102 do not interfere with thepatient, physician, anesthesiologist, or any other equipment. Duringprocedures, a user may control the robotic arms 102 using controldevices such as the command console.

In some embodiments, the robotic arm 102 includes set up joints that usea combination of brakes and counter-balances to maintain a position ofthe robotic arm 102. The counter-balances may include gas springs orcoil springs. The brakes, e.g., fail safe brakes, may be includemechanical and/or electrical components. Further, the robotic arms 102may be gravity-assisted passive support type robotic arms.

Each robotic arm 102 may be coupled to a robotic manipulator like aninstrument device manipulator (IDM) 117 using a mechanism changerinterface (MCI) 116. The IDM 117 can be removed and replaced with adifferent type of IDM, for example, a first type of IDM manipulates anendoscope, while a second type of IDM manipulates a laparoscope. The MCI116 includes connectors to transfer pneumatic pressure, electricalpower, electrical signals, and optical signals from the robotic arm 102to the IDM 117. The MCI 116 can be a set screw or base plate connector.The IDM 117 manipulates surgical instruments such as the endoscope 118using techniques including direct drive, harmonic drive, geared drives,belts and pulleys, magnetic drives, and the like. The MCI 116 isinterchangeable based on the type of IDM 117 and can be customized for acertain type of surgical procedure. The robotic 102 arm can include ajoint level torque sensing and a wrist at a distal end, such as the KUKAAG® LBR5 robotic arm. The endoscope 118 is a tubular and flexiblesurgical instrument that is inserted into the anatomy of a patient tocapture images of the anatomy (e.g., body tissue). In particular, theendoscope 118 includes one or more imaging devices (e.g., cameras orother types of imaging sensors) that capture the images. The imagingdevices may include one or more optical components such as an opticalfiber, fiber array, or lens. The optical components move along with thetip of the endoscope 118 such that movement of the tip of the endoscope118 results in changes to the images captured by the imaging devices.The endoscope 118 is further described with reference to FIGS. 3A-4B inSection IV. Endoscope.

Robotic arms 102 of the surgical robotic system 100 manipulate theendoscope 118 using elongate movement members. The elongate movementmembers may include pull wires, also referred to as pull or push wires,cables, fibers, or flexible shafts. For example, the robotic arms 102actuate multiple pull wires coupled to the endoscope 118 to deflect thetip of the endoscope 118. The pull wires may include both metallic andnon-metallic materials such as stainless steel, Kevlar, tungsten, carbonfiber, and the like. The endoscope 118 may exhibit nonlinear behavior inresponse to forces applied by the elongate movement members. Thenonlinear behavior may be based on stiffness and compressibility of theendoscope 118, as well as variability in slack or stiffness betweendifferent elongate movement members.

FIGS. 1B-1F show various perspective views of the surgical roboticsystem 100 coupled to a robotic platform 150 (or surgical bed),according to various embodiments. Specifically, FIG. 1B shows a sideview of the surgical robotic system 100 with the robotic arms 102manipulating the endoscopic 118 to insert the endoscopic inside apatient's body, and the patient is lying on the robotic platform 150.FIG. 1C shows a top view of the surgical robotic system 100 and therobotic platform 150, and the endoscopic 118 manipulated by the roboticarms is inserted inside the patient's body. FIG. 1D shows a perspectiveview of the surgical robotic system 100 and the robotic platform 150,and the endoscopic 118 is controlled to be positioned horizontallyparallel with the robotic platform. FIG. 1E shows another perspectiveview of the surgical robotic system 100 and the robotic platform 150,and the endoscopic 118 is controlled to be positioned relativelyperpendicular to the robotic platform. In more detail, in FIG. 1E, theangle between the horizontal surface of the robotic platform 150 and theendoscopic 118 is 75 degree. FIG. 1F shows the perspective view of thesurgical robotic system 100 and the robotic platform 150 shown in FIG.1E, and in more detail, the angle between the endoscopic 118 and thevirtual line 160 connecting one end 180 of the endoscopic and therobotic arm 102 that is positioned relatively farther away from therobotic platform is 90 degree.

II. Command Console

FIG. 2 shows an example command console 200 for the example surgicalrobotic system 100, according to one embodiment. The command console 200includes a console base 201, display modules 202, e.g., monitors, andcontrol modules, e.g., a keyboard 203 and joystick 204. In someembodiments, one or more of the command console 200 functionality may beintegrated into a base 101 of the surgical robotic system 100 or anothersystem communicatively coupled to the surgical robotic system 100. Auser 205, e.g., a physician, remotely controls the surgical roboticsystem 100 from an ergonomic position using the command console 200.

The console base 201 may include a central processing unit, a memoryunit, a data bus, and associated data communication ports that areresponsible for interpreting and processing signals such as cameraimagery and tracking sensor data, e.g., from the endoscope 118 shown inFIG. 1. In some embodiments, both the console base 201 and the base 101perform signal processing for load-balancing. The console base 201 mayalso process commands and instructions provided by the user 205 throughthe control modules 203 and 204. In addition to the keyboard 203 andjoystick 204 shown in FIG. 2, the control modules may include otherdevices, for example, computer mice, trackpads, trackballs, controlpads, video game controllers, and sensors (e.g., motion sensors orcameras) that capture hand gestures and finger gestures.

The user 205 can control a surgical instrument such as the endoscope 118using the command console 200 in a velocity mode or position controlmode. In velocity mode, the user 205 directly controls pitch and yawmotion of a distal end of the endoscope 118 based on direct manualcontrol using the control modules. For example, movement on the joystick204 may be mapped to yaw and pitch movement in the distal end of theendoscope 118. The joystick 204 can provide haptic feedback to the user205. For example, the joystick 204 vibrates to indicate that theendoscope 118 cannot further translate or rotate in a certain direction.The command console 200 can also provide visual feedback (e.g., pop-upmessages) and/or audio feedback (e.g., beeping) to indicate that theendoscope 118 has reached maximum translation or rotation.

In position control mode, the command console 200 uses athree-dimensional (3D) map of a patient and pre-determined computermodels of the patient to control a surgical instrument, e.g., theendoscope 118. The command console 200 provides control signals torobotic arms 102 of the surgical robotic system 100 to manipulate theendoscope 118 to a target location. Due to the reliance on the 3D map,position control mode requires accurate mapping of the anatomy of thepatient.

In some embodiments, users 205 can manually manipulate robotic arms 102of the surgical robotic system 100 without using the command console200. During setup in a surgical operating room, the users 205 may movethe robotic arms 102, endoscopes 118, and other surgical equipment toaccess a patient. The surgical robotic system 100 may rely on forcefeedback and inertia control from the users 205 to determine appropriateconfiguration of the robotic arms 102 and equipment.

The display modules 202 may include electronic monitors, virtual realityviewing devices, e.g., goggles or glasses, and/or other means of displaydevices. In some embodiments, the display modules 202 are integratedwith the control modules, for example, as a tablet device with atouchscreen. Further, the user 205 can both view data and input commandsto the surgical robotic system 100 using the integrated display modules202 and control modules.

The display modules 202 can display 3D images using a stereoscopicdevice, e.g., a visor or goggle. The 3D images provide an “endo view”(i.e., endoscopic view), which is a computer 3D model illustrating theanatomy of a patient. The “endo view” provides a virtual environment ofthe patient's interior and an expected location of an endoscope 118inside the patient. A user 205 compares the “endo view” model to actualimages captured by a camera to help mentally orient and confirm that theendoscope 118 is in the correct—or approximately correct—location withinthe patient. The “endo view” provides information about anatomicalstructures, e.g., the shape of an intestine or colon of the patient,around the distal end of the endoscope 118. The display modules 202 cansimultaneously display the 3D model and computerized tomography (CT)scans of the anatomy the around distal end of the endoscope 118.Further, the display modules 202 may overlay the already determinednavigation paths of the endoscope 118 on the 3D model and CT scans.

In some embodiments, a model of the endoscope 118 is displayed with the3D models to help indicate a status of a surgical procedure. Forexample, the CT scans identify a lesion in the anatomy where a biopsymay be necessary. During operation, the display modules 202 may show areference image captured by the endoscope 118 corresponding to thecurrent location of the endoscope 118. The display modules 202 mayautomatically display different views of the model of the endoscope 118depending on user settings and a particular surgical procedure. Forexample, the display modules 202 show an overhead fluoroscopic view ofthe endoscope 118 during a navigation step as the endoscope 118approaches an operative region of a patient.

III. Instrument Device Manipulator

FIG. 3A shows an isometric view of an example independent drivemechanism of the IDM 117 shown in FIG. 1, according to one embodiment.The independent drive mechanism can tighten or loosen the pull wires321, 322, 323, and 324 (e.g., independently from each other) of anendoscope by rotating the output shafts 305, 306, 307, and 308 of theIDM 117, respectively. Just as the output shafts 305, 306, 307, and 308transfer force down pull wires 321, 322, 323, and 324, respectively,through angular motion, the pull wires 321, 322, 323, and 324 transferforce back to the output shafts. The IDM 117 and/or the surgical roboticsystem 100 can measure the transferred force using a sensor, e.g., astrain gauge further described below.

FIG. 3B shows a conceptual diagram that shows how forces may be measuredby a strain gauge 334 of the independent drive mechanism shown in FIG.3A, according to one embodiment. A force 331 may direct away from theoutput shaft 305 coupled to the motor mount 333 of the motor 337.Accordingly, the force 331 results in horizontal displacement of themotor mount 333. Further, the strain gauge 334 horizontally coupled tothe motor mount 333 experiences strain in the direction of the force331. The strain may be measured as a ratio of the horizontaldisplacement of the tip 335 of strain gauge 334 to the overallhorizontal width 336 of the strain gauge 334.

In some embodiments, the IDM 117 includes additional sensors, e.g.,inclinometers or accelerometers, to determine an orientation of the IDM117. Based on measurements from the additional sensors and/or the straingauge 334, the surgical robotic system 100 can calibrate readings fromthe strain gauge 334 to account for gravitational load effects. Forexample, if the IDM 117 is oriented on a horizontal side of the IDM 117,the weight of certain components of the IDM 117 may cause a strain onthe motor mount 333. Accordingly, without accounting for gravitationalload effects, the strain gauge 334 may measure strain that did notresult from strain on the output shafts.

IV. Endoscope

FIG. 4A shows a top view of an example endoscope 118, according to oneembodiment. The endoscope 118 includes a leader 415 tubular componentnested or partially nested inside and longitudinally-aligned with asheath 411 tubular component. The sheath 411 includes a proximal sheathsection 412 and distal sheath section 413. The leader 415 has a smallerouter diameter than the sheath 411 and includes a proximal leadersection 416 and distal leader section 417. The sheath base 414 and theleader base 418 actuate the distal sheath section 413 and the distalleader section 417, respectively, for example, based on control signalsfrom a user of a surgical robotic system 100. The sheath base 414 andthe leader base 418 are, e.g., part of the IDM 117 shown in FIG. 1.

Both the sheath base 414 and the leader base 418 include drivemechanisms (e.g., the independent drive mechanism further described withreference to FIG. 3A-B in Section III. Instrument Device Manipulator) tocontrol pull wires coupled to the sheath 411 and leader 415. Forexample, the sheath base 414 generates tensile loads on pull wirescoupled to the sheath 411 to deflect the distal sheath section 413.Similarly, the leader base 418 generates tensile loads on pull wirescoupled to the leader 415 to deflect the distal leader section 417. Boththe sheath base 414 and leader base 418 may also include couplings forthe routing of pneumatic pressure, electrical power, electrical signals,or optical signals from IDMs to the sheath 411 and leader 414,respectively. A pull wire may include a steel coil pipe along the lengthof the pull wire within the sheath 411 or the leader 415, whichtransfers axial compression back to the origin of the load, e.g., thesheath base 414 or the leader base 418, respectively.

The endoscope 118 can navigate the anatomy of a patient with ease due tothe multiple degrees of freedom provided by pull wires coupled to thesheath 411 and the leader 415. For example, four or more pull wires maybe used in either the sheath 411 and/or the leader 415, providing eightor more degrees of freedom. In other embodiments, up to three pull wiresmay be used, providing up to six degrees of freedom. The sheath 411 andleader 415 may be rotated up to 360 degrees along a longitudinal axis406, providing more degrees of motion. The combination of rotationalangles and multiple degrees of freedom provides a user of the surgicalrobotic system 100 with a user friendly and instinctive control of theendoscope 118.

FIG. 4B illustrates an example endoscope tip 430 of the endoscope 118shown in FIG. 4A, according to one embodiment. In FIG. 4B, the endoscopetip 430 includes an imaging device 431 (e.g., a camera), illuminationsources 432, and one or more electromagnetic (EM) coils 434 (alsoreferred to as EM sensors). The illumination sources 432 provide lightto illuminate an interior portion of an anatomical space. The providedlight allows the imaging device 431 to record images of that space,which can then be transmitted to a computer system such as commandconsole 200 for processing as described herein. EM coils 434 located onthe tip 430 may be used with an electromagnetic tracking system todetect the position and orientation of the endoscope tip 430 while it isdisposed within an anatomical system. In some embodiments, the coils maybe angled to provide sensitivity to electromagnetic fields alongdifferent axes, giving the ability to measure a full 6 degrees offreedom: three positional and three angular. In other embodiments, onlya single coil may be disposed within the endoscope tip 430, with itsaxis oriented along the endoscope shaft of the endoscope 118; due to therotational symmetry of such a system, it is insensitive to roll aboutits axis, so only 5 degrees of freedom may be detected in such a case.The endoscope tip 430 further comprises a working channel 436 throughwhich surgical instruments, such as biopsy needles, may be insertedalong the endoscope shaft, allowing access to the area near theendoscope tip.

The principles described in the above application are also applicable tocatheter designs. Generally, although the preceding and followingsections of this description describe endoscope embodiments, this ismerely one example, and the description that follows can also beimplemented and/or used in conjunction with catheters as well, or moregenerally any flexible instrument comprising an elongate body.

V. Registration Transform of Em System to 3D Model

V. A. Schematic Setup of an Em Tracking System

FIG. 5A shows an example schematic setup of an EM tracking system 505included in a surgical robotic system 500, according to one embodiment.In FIG. 5A, multiple robot components (e.g., window field generator,reference sensors as described below) are included in the EM trackingsystem 505. The robotic surgical system 500 includes a surgical bed 511to hold a patient's body. Beneath the bed 511 is the window fieldgenerator (WFG) 512 configured to sequentially activate a set of EMcoils (e.g., the EM coils 434 shown in FIG. 4B). The WFG 512 generatesan alternating current (AC) magnetic field over a wide volume; forexample, in some cases it may create an AC field in a volume of about0.5×0.5×0.5 m.

Additional fields may be applied by further field generators to aid intracking instruments within the body. For example, a planar fieldgenerator (PFG) may be attached to a system arm adjacent to the patientand oriented to provide an EM field at an angle. Reference sensors 513may be placed on the patient's body to provide local EM fields tofurther increase tracking accuracy. Each of the reference sensors 513may be attached by cables 514 to a command module 515. The cables 514are connected to the command module 515 through interface units 516which handle communications with their respective devices as well asproviding power. The interface unit 516 is coupled to a system controlunit (SCU) 517 which acts as an overall interface controller for thevarious entities mentioned above. The SCU 517 also drives the fieldgenerators (e.g., WFG 512), as well as collecting sensor data from theinterface units 516, from which it calculates the position andorientation of sensors within the body. The SCU 517 may be coupled to apersonal computer (PC) 518 to allow user access and control.

The command module 515 is also connected to the various IDMs 519 coupledto the surgical robotic system 500 as described herein. The IDMs 519 aretypically coupled to a single surgical robotic system (e.g., thesurgical robotic system 500) and are used to control and receive datafrom their respective connected robotic components; for example, roboticendoscope tools or robotic arms. As described above, as an example, theIDMs 519 are coupled to an endoscopic tool (not shown here) of thesurgical robotic system 500

The command module 515 receives data passed from the endoscopic tool.The type of received data depends on the corresponding type ofinstrument attached. For example, example received data includes sensordata (e.g., image data, EM data), robot data (e.g., endoscopic and IDMphysical motion data), control data, and/or video data. To better handlevideo data, a field-programmable gate array (FPGA) 520 may be configuredto handle image processing. Comparing data obtained from the varioussensors, devices, and field generators allows the SCU 517 to preciselytrack the movements of different components of the surgical roboticsystem 500, and for example, positions and orientations of thesecomponents.

In order to track a sensor through the patient's anatomy, the EMtracking system 505 may require a process known as “registration,” wherethe system finds the geometric transformation that aligns a singleobject between different coordinate systems. For instance, a specificanatomical site on a patient has two different representations in the 3Dmodel coordinates and in the EM sensor coordinates. To be able toestablish consistency and common language between these two differentcoordinate systems, the EM tracking system 505 needs to find thetransformation that links these two representations, i.e., registration.For example, the position of the EM tracker relative to the position ofthe EM field generator may be mapped to a 3D coordinate system toisolate a location in a corresponding 3D model.

V. C. On-the-Fly Electromagnetic Registration

FIGS. 5B-5E show example graphs 510-540 illustrating on-the-flyregistration of an EM system to a 3D model of a path through a tubularnetwork, according to one embodiment. The navigation configurationsystem described herein allows for on-the-fly registration of the EMcoordinates to the 3D model coordinates without the need for independentregistration prior to an endoscopic procedure. In more detail, FIG. 5Ashows that the coordinate systems of the EM tracking system and the 3Dmodel are initially not registered to each other, and the graph 530 inFIG. 5A shows the registered location of an endoscope tip 531 movingalong a planned navigation path 532 through a branched tubular network(not shown here), and the location of the instrument tip 531 as well asthe planned path 532 are derived from the 3D model. The location of theendoscopic tip 531 shown in FIG. 5B is an expected location derived fromthe 3D model rather than the actual location of the tip. The actualposition of the tip is repeatedly measured by the EM tracking system505, resulting in multiple measured location data points 533 based on EMdata. As shown in FIG. 5B, the data points 533 derived from EM trackingare initially located far from the position of the endoscope tip 531expected from the 3D model, reflecting the lack of registration betweenthe EM coordinates and the 3D model coordinates. There may be severalreasons for this, for example, even if the endoscope tip is being movedrelatively smoothly through the tubular network, there may still be somevisible scatter in the EM measurement, due to breathing movement of thelungs of the patient.

The points on the 3D model may also be determined and adjusted based oncorrelation between the 3D model itself, image data received fromimaging sensors (e.g., cameras) and robot data from robot commands. The3D transformation between these points and collected EM data points willdetermine the initial registration of the EM coordinate system to the 3Dmodel coordinate system.

FIG. 5C shows a graph 540 at a later temporal stage compared with thegraph 530, according to one embodiment. More specifically, the graph 540shows the endoscope tip 531 expected from the 3D model has been movedfarther along the preplanned navigation path 532, as illustrated by theshift from the original position of the instrument tip 531 shown in FIG.5B along the path to the position shown in FIG. 5C. During the EMtracking between generation of the graph 530 and generation of graph540, additional data points 533 have been recorded by the EM trackingsystem but the registration has not yet been updated based on the newlycollected EM data. As a result, the data points 533 in FIG. 5C areclustered along a visible path 544, but that path differs in locationand orientation from the planned navigation path 532 the endoscope tipis being directed by the operator to travel along. Eventually, oncesufficient data (e.g., EM data) is accumulated, compared with using onlythe 3D model or only the EM data, a relatively more accurate estimatecan be derived from the transform needed to register the EM coordinatesto those of the 3D model. The determination of sufficient data may bemade by threshold criteria such as total data accumulated or number ofchanges of direction. For example, in a branched tubular network such asa bronchial tube network, it may be judged that sufficient data havebeen accumulated after arriving at two branch points.

FIG. 5D shows a graph 550 shortly after the navigation configurationsystem has accumulated a sufficient amount of data to estimate theregistration transform from EM to 3D model coordinates, according to oneembodiment. The data points 533 in FIG. 5D have now shifted from theirprevious position as shown in FIG. 5C as a result of the registrationtransform. As shown in FIG. 5D, the data points 533 derived from EM datais now falling along the planned navigation path 532 derived from the 3Dmodel, and each data point among the data points 533 is now reflecting ameasurement of the position of endoscope tip 531 in the coordinatesystem of the 3D model, and as introduced above in FIGS. 5B-5C, theposition of the tip 531 shown in FIG. 5D is an expected location derivedfrom the 3D model rather than an actual location of the tip. In someembodiments, as further data are collected, the registration transformmay be updated to increase accuracy. In some cases, the data used todetermine the registration transformation may be a subset of data chosenby a moving window, so that the registration may change over time, whichgives the ability to account for changes in the relative coordinates ofthe EM and 3D models—for example, due to movement of the patient.

FIG. 5E shows an example graph 560 in which the registered endoscope tip531 has reached the end of the planned navigation path 532, arriving atthe target location in the tubular network, according to one embodiment.As introduced above in FIGS. 5B-5D, the position of the tip 531 shown inFIG. 5E is an expected location derived from the 3D model rather than anactual location of the tip. As shown in FIG. 5E, the recorded EM datapoints 533 is now generally tracks along the planned navigation path532, which represents the tracking of the endoscope tip throughout theprocedure. Each data point reflects a transformed location due to theupdated registration of the EM tracking system to the 3D model.

In some embodiments, each of the graphs shown in FIGS. 5B-5E can beshown sequentially on a display visible to a user as the endoscope tipis advanced in the tubular network. In some embodiments, the processorcan be configured with instructions from the navigation configurationsystem such that the model shown on the display remains substantiallyfixed when the measured data points are registered to the display byshifting of the measured path shown on the display in order to allow theuser to maintain a fixed frame of reference and to remain visuallyoriented on the model and on the planned path shown on the display.

VI. Lower Body Surgery

FIGS. 6A-6B show different views of a surgical robotics system withcolumn-mounted arms configured to access the lower body area of apatient according to one embodiment. More specifically, FIG. 6A shows aperspective view of a surgical robotics system 600 with column-mountedarms configured to access the lower body area of a patient 608 accordingto one embodiment. The surgical robotics system 600 includes a set ofrobotic arms (including five robotic arms in total) and a set of threecolumn rings. A first robotic arm 670A and a second robotic arm 670B arecoupled to a first column ring 605A. A third robotic arm 670C and afourth robotic arm 670D are coupled to a second column ring 605B. Afifth robotic arm 670E is coupled to a third column ring 605C. FIG. 6Ashows a wireframe of the patient 608 lying on the table 601 undergoing asurgical procedure, e.g., ureteroscopy, involving access to the lowerbody area of the patient 608. Legs of the patient 608 are not shown inorder to avoid obscuring portions of the surgical robotics system 600.

The surgical robotics system 600 configures the set of robotic arms toperform a surgical procedure on the lower body area of the patient 608.Specifically, the surgical robotics system 600 configures the set ofrobotic arms to manipulate a surgical instrument 610. The set of roboticarms insert the surgical instrument 610 along a virtual rail 690 intothe groin area of the patient 608. Generally, a virtual rail 690 is aco-axial trajectory along which the set of robotic arms translates asurgical instrument (e.g., a telescoping instrument). The second roboticarm 670B, the third robotic arm 670C, and the fifth robotic arm 670E arecoupled, e.g., holding, the surgical instrument 610. The first roboticarm 670A and the fourth robotic arm 670D are stowed to the sides of thesurgical robotics system because they are not necessarily required tofor the surgical procedure—or at least part of the surgicalprocedure—shown in FIG. 6A. The robotic arms are configured such thatthey manipulate the surgical instrument 610 from a distance away fromthe patient 608. This is advantageous, for example, because there isoften limited space available closer toward the patient's body or thereis a sterile boundary around the patient 608. Further, there may also bea sterile drape around surgical equipment. During a surgical procedure,only sterile objects are allowed pass the sterile boundary. Thus, thesurgical robotics system 600 may still use robotic arms that arepositioned outside of the sterile boundary and that are covered withsterilized drapes to perform a surgical procedure.

In one embodiment, the surgical robotics system 600 configures the setof robotic arms to perform an endoscopy surgical procedure on thepatient 608. The set of robotic arms hold an endoscope, e.g., thesurgical instrument 610. The set of robotic arms insert the endoscopeinto the patient's body via an opening in the groin area of the patient608. The endoscope is a flexible, slender, and tubular instrument withoptical components such as a camera and optical cable. The opticalcomponents collect data representing images of portions inside thepatient's body. A user of the surgical robotics system 600 uses the datato assist with performing the endoscopy.

FIG. 6B is a top view of the surgical robotics system 600 withcolumn-mounted arms configured to access the lower body area of thepatient 608 according to one embodiment.

VII. Basket Apparatus

FIGS. 7A-7B show a side view of a basket apparatus 700 of a surgicalrobotic system and how the basket may be used to capture a kidney stone,according to one embodiment. More specifically, FIG. 7A shows a sideview of the basket apparatus 700, according to one embodiment. FIGS. 7Band 7C show how the basket apparatus 700 may be used to capture anobject, such as a urinary stone, according to one embodiment.

The robotically steerable basket apparatus 700 may be operatively andremovably coupled to any of the IDMs described herein and above, such asIDM 117 described above. The robotically steerable basket apparatus 700may be advanced through a natural or artificially created orifice in asubject or patient to capture a target object within the body of thesubject or patient. For instance, the robotically steerable basketapparatus 700 may be advanced with the surgical robotic system 100through the urethra, and optionally the bladder, ureter, and/or thekidney to capture a kidney stone (ST). As another example, therobotically steerable basket apparatus 700 may be advanced into thegallbladder to capture a gallstone. In some embodiments, the roboticallysteerable basket apparatus 700 may be advanced through another workingchannel of a catheter, ureteroscope, endoscope, or similar device (e.g.,within a 1.2 mm diameter working channel.)

The robotically steerable basket apparatus 700 may include a handle ortool base 760 adapted to removably and operatively couple with the IDM117. The tool base 760 may include a number of capstans 720 to couple tothe output shafts or drive units of the IDM so that the IDM can actuatethe capstans 720 as well as other actuation elements coupled thereto.The basket apparatus 700 further includes a number of pull wires (alsoreferred to as tendons) 730. The pull wires 730 are coupled to thecapstans 720 at one end. The pull wires 730 run straight along the longaxis of the apparatus 700, and are prevented from sagging or twisting byan elongate support shaft 740. The elongate support shaft 740 includes aplurality of lumens and channels through which the pull wires 730 maytraverse along the direction of the long axis of the apparatus 700. Theelongate support shaft 740 may be flexible to facilitate advancement ofthe basket apparatus 700 through a tortuous tissue tract or bodilychannel, such as the urethra and ureter.

The pull wires 730 may be coupled to one another at the distal-most tip752 of the basket apparatus 700. For example, the basket apparatus 700may include two different pairs of pull wires 730, with each pull wirepair forming a loop with the tips of the loops coupled to one another attip 752 and each pull wire having its two ends threaded through oppositeperipheral channels or lumens of the elongate support shaft 740. The twotips of the looped pull wires may be coupled together in any number ofways. For example, they may be soldered together, crimped together,braided together, bonded together with an adhesive, tied together with asuture or other thread, etc. Once connected together, each pair of pullwires forming a loop can also be referred to as a single pull wire, ifthat terminology is preferred in a particular implementation.

When the tool base 760 is coupled to an IDM, the capstans 720 mayactuate the pull wires 730 so that the pull wires 730 can be translatedproximally or distally in the axial (long axis) direction, such asrelative to the elongate support shaft 740. One or more of the pull wire730 may be translated independently from one another, such as by theirrespective capstans 720.

The distal ends of the pull wires 730 may extend from the distal end 744of the elongate support shaft 740 to form a distal wire basket 750. Thedistal ends of the pull wires 730 may be retracted by the capstans 720located at the proximal end of the elongate support shaft 740 tocollapse the basket 750 into the elongate support shaft 740. Retractionof the basket 750 into the elongate support shaft 740 can lower theprofile of the basket apparatus 700 to facilitate the advancement of thebasket apparatus 700 into a tissue tract or bodily channel. Conversely,the capstans 720 may be actuated to extend the pull wires 730 out fromthe elongate support shaft 740 so that the basket 750 may expand. Forinstance, once the distal end 744 of the elongate support shaft 740 ispositioned near a stone ST, the basket 750 may be expanded to capturethe stone ST.

The basket 750 may be extended from elongate support shaft 740 atdifferent amounts of extension to vary the size of the basket 750. Forinstance, as illustrated in FIGS. 7B and 7C, the basket 750 mayinitially be extended to an enlarged size to capture the stone ST withinthe basket 750 and then the basket 750 may be partially collapsed (i.e.,reduced in size) to secure the stone within the basket 750. As furthershown in FIG. 7B, the pull wires 730 may be selectively actuated tosteer or tip the basket 550 to facilitate capture of the stone ST. Theelongate support shaft 540 may be held stationary relative to the pullwires 530 while the pull wires 530 are differentially actuated. Thebasket 750 may be steered in any number of directions by thedifferential actuation of the individual pull wires 730 such that it hasa 360° range of motion. For example, one end of an individual pull wire730 may be held stationary while the other end is pulled or pushed totip the basket 750 toward or away from the moving end, respectively. Inother examples, the individual ends of the pull wires 730 may bedifferentially pulled, pushed, or held stationary to vary the degreeand/or direction of the tipping.

The degree of movement of the capstans 720 may be indicative of thedegree and/or direction of the tipping of the basket 750 and also of itscurrent size. Therefore, in some embodiments, the robotic system, andthe IDM in particular, can determine and/or track the currentconfiguration of the basket 750 positioned within a subject or patient'sbody based on the feedback or information from the capstans 720, thedrive unit(s), or output shaft(s) and without visualization of thebasket 750. Alternatively or in combination, the basket 750 may bevisualized to determine and/or track its current configuration. The pullwires 730 may be formed from a shape memory material or metal (e.g., aNickel-Titanium alloy such as Nitinol) so that the distal ends of thepull wires 730 may be biased to assume the basket shape whenunconstrained and/or at body temperature.

VIII. Object Sizing System

FIG. 8 shows an example block diagram of an object sizing system 800,according to one embodiment. In FIG. 8, the object sizing system 800includes multiple data stores including input data stores such as animage data store 852, a robot data store 854, and an output data store,a size data store 858. The object sizing system 800 also includesmultiple modules.

The interface module 810 receives various of input data from inputdevices (EM sensor, imaging sensor, IDMs) of the surgical robotic system100. The interface module 810 may also provide the output data (e.g.,estimated size of the target object) to a user.

The navigation module 820 navigates the endoscopic tool through thetubular network of the patient's body to arrive at an anatomical sitewithin a lumen of the tubular network, for example, near the positionwhere the target object locates. The navigation module 820 alsoinstructs the endoscopic tip to be positioned with a particularorientation proximal to the target object. A sequence of images of theobject are captured by the imaging sensor while the endoscopic tip ispositioned in a particular location with a specific orientation proximalto the target object. The navigation module 820 navigates the endoscopictool to arrive at different positions with different orientationsproximal to the target object, and images of the object are capturedwhile the endoscopic tool is at each of those positions andorientations.

The image analysis module 830 analyzes the images of the object capturedby the imaging sensor when the endoscopic tool is located at differentpositions with specific orientations proximal to the target object. Inmore detail, the image analysis module 830 extracts various input data(e.g., image data, robot data, EM data) stored in the input data stores852-856 and employs various image analysis techniques (e.g., imagestitching techniques, optical flow techniques) for estimating a size ofthe target object, as more fully described below with reference to FIGS.10-13.

The size estimation module 840 determines an estimated size of thetarget object based on the analysis performed by the image analysismodule 830, and stores the size data in the size data store 858.

The block diagram of the object sizing system 800 shown in FIG. 8 ismerely one example, and in alternative embodiments not shown, the objectsizing system 800 can include different and/or addition entities.Likewise, functions performed by various entities of the system 800 maydiffer according to different embodiments.

The input data, as used herein, refers to raw data gathered from and/orprocessed by input devices (e.g., command module, imaging sensor, EMsensor, IDM) for generating an estimated size of the target object. Eachtype of the input data stores 852-856 stores the name-indicated type ofdata for access and use by the multiple modules 810-840.

Image data may include one or more image frames captured by the imagingsensor at the instrument tip, as well as information such as sequence ofthe captured images, frame rates or timestamps that allow adetermination of the time elapsed between pairs of frames. The size datastore 858 receives and stores size data provided by the size estimationmodule 840. The size data describes an estimated size of the targetobject. Example size data includes size data like absolute outer axialdiameter of the object. Example size data also includes size datadescribing size of the object in relation to environment surrounding theobject, e.g., size of the object compared with diameter of lumen (e.g.,ureter) and/or basket capturing the object. The estimated size may, forexample, also indicate whether or not the object fits in the basket. Theestimated size of the object may also be described in relation to shapeof the object. For example, for a round object like a stone, theestimated size of the object indicates outer axial diameter of theobject from all directions.

Robot data includes data related to physical movement of the endoscopictool or part of the endoscopic tool (e.g., the endoscopic tip or sheath)within the lumen. Example robot data includes both command data andactual motion data. The command data describes commands by the surgicalrobotic system for instructing the endoscopic tip to reach a specificanatomical site and/or change its orientation (e.g., motion of one ormore pull wires, tendons or shafts of the endoscope), and can beprovided by the IDMs. The actual motion data describes actual motion ofthe endoscopic tip, and can be provided by EM sensors or deduced fromother data such as image data collected from the imaging sensor at thedistal tip of the endoscope (the endoscopic tip).

For the purposes of the techniques described herein, in one embodiment,the actual motion data provides the same information about positions andmovements of the endoscopic tip as provided by the command datadescribing the commands that instructs the tip to reach these positionsand to achieve these movements. In another embodiment, the actual motiondata provides different information about positions and movements of thetip from the corresponding command data, as the endoscopic tip may notstrictly follow corresponding commands. In this case, actual motion datamay be used to calibrate command data and/or otherwise providesupplemental data that helps verify the motion of the endoscopic tip.

Regardless of whether command data or actual motion data in practice,both types of data are capable of providing information that describesinsertion, position, and articulation of the endoscopic tool during thecourse of a procedure. As one example, the information provided caninclude insertion information indicating the depth the endoscopic toolas a whole inside a lumen or inside a patient's body, positioninformation indicating the position of the endoscopic tip relative tothe object, and articulation information indicating orientation (e.g.,pitch, raw, and yaw) of the endoscopic tip in relation to the object.

The object sizing system 800 may further use any information itgenerates to further provide information to an operator and/or generaterecommendations based on that information to provide to an operator.This may include, for example, information about whether the basket canfit the object based on their relative sizes, whether the object willfit down the working channel of the instrument or through a percutaneousport in a patient, whether the object should be broken before removal,etc.

VIII. A. GENERAL OBJECT SIZING PROCESS

FIG. 9 shows an example flowchart 900 illustrating a process ofestimating size of an object within a lumen of a patient's body by theobject sizing system shown in FIG. 8, according to one embodiment. Asshown in FIG. 9, initially, the object sizing system 800 identifies 910a presence of an object (e.g., a kidney stone) positioned within a lumenof a patient. Upon identification of the object, the object sizingsystem 800 navigates 920 an endoscope into the lumen to arrive at aspecific location with a specific orientation proximal to the object.The object sizing system 800 instructs the imaging sensor to capture 930one or more sequential images of the object at the specific locationwith the specific orientation. The object sizing system 800 alsoacquires 940 a set of robot data associated with the captured images.Specifically, for each of the captured images, there is robot datadescribing the endoscope motion as well as position measurements of theendoscope and the object for that image.

If the captured images are inadequate 958 to determine the object size,the object sizing system 800 navigates 920 the endoscope to a differentlocation with a specific orientation proximal to the object, andinstructs the imaging sensor to capture 930 one or more additionalsequential images of the object at that location with the specificorientation. As described below, the object sizing system 800 is capableof estimating a size of the object with improved accuracy with imagescaptured at different positions with same or different orientationsproximal to the object, compared with using images at only a singlelocation.

After sufficient images of the object are captured 954 with theendoscope at multiple locations and/or orientations in relation to theobject, the object sizing system 800 analyzes 960 the captured imagesbased on the acquired robot data. The object sizing system 800 estimates970 a size of the object based on the analysis of the captured images.

VIII. B. Object Size Estimation Techniques using Image Analysis

VIII. B. 1 Image Stitching Techniques

FIG. 10 shows an example flowchart illustrating a process of estimatingsize of an object using image stitching techniques, according to oneembodiment. As introduced above, the object sizing system 800 identifies1010 a presence of an object positioned within a lumen of a patient,.

The object sizing system 800 also acquires 1020 a sequence of capturedimages with robot data and EM data associated with the captured images.In one embodiment, the images are captured with the endoscope(specifically an imaging sensor coupled to the endoscopic tip) moved tobe positioned at different locations in relation to the object. As oneexample, the object sizing system 800 instructs the endoscope to move todifferent depths in the lumen where the object locates, as representedby insertion data. As another example, at a fixed or varied depth intothe lumen, the object sizing system 800 instructs the imaging sensor ofthe endoscope to be positioned at different locations (e.g., left,right) relative to the object, as represented by position measurements.As a further example, at a fixed or varied depth in the lumen and at afixed or varied location relative to the object, the object sizingsystem 800 instructs the imaging sensor to articulate around the object,and as represented articulation data. The robot data acquired such asthe insertion data, position measurements, and articulation datadescribes these various motion of the endoscope and the imaging sensoraccordingly in relation to the object, which allows the object sizingsystem 800 to derive a complete understanding of the various motions ofthe imaging sensor and thereby deduce an accurate estimate of the objectsize.

The object sizing system applies 1030 image stitching techniques to thecaptured images based on the acquired image sequence along with therobot and/or EM data received for the periods of time when the imagesequence was acquired. The object sizing system 800 is capable ofstitching multiple captured images together to create a 3D model of theobject. As one example for stitching two images (e.g., Image A and ImageB) together, the object sizing system 800 acquires EM and/or robot dataassociated with the captured images. This data indicates positions ofthe object shown in these two images, for example, spatial measurementsof the object like x, y, z (Cartesian) coordinates, as well asorientation (angular coordinates) of the camera while the camera iscapturing those images. More details about the EM data and robot datathat can provide these coordinates can be found in patent applicationSer. No. 15/268,238 that is hereby incorporated by reference in itsentirety. The object sizing system 800 identifies overlapped portionbetween Image A and Image B based on the EM data and/or robot data, andstitches these two images together to create a combined image based onthe identified overlapped portion. To create a 3D model of the object,the object sizing system 800 may stitch multiple images taken fromdifferent views of the object. Example stitching algorithms include 3Dstitching, normalized stitching, least squares matching, normalizedcross correlation and stitching algorithms based on normalized mutualinformation.

The object sizing system identifies 1040 the structure based on thecreated 3D model. The structure may include a shape and a size of theobject. The identified structure can be a 3D shape of the object, andexample metrics of the 3D shape include longest axis and/or shortestaxis of the object and other metrics. For example, the identifiedstructure may include an approximate radius or outer axial diameter orradius of a round kidney stone. The object sizing system 800 determines1050 an estimated size of the object based on the identified structure.

VIII. B.2 Basketing Techniques

VIII. B. 2.1 Basket Marker Detection

FIG. 11A shows an example flowchart illustrating a process of estimatingsize of an object using a basketing apparatus and basket markerdetection, according to one embodiment. The object sizing system 800instructs a basket apparatus to capture 1110 an object within a basket.For example, the basket apparatus can be the basket apparatus 700 shownin FIGS. 7A-7C, and the object is an object (e.g., a kidney stone)within a lumen of a patient's body.

After receiving 1120 notification that the object is captured within thebasket, the object sizing system 800 performs 1130 analysis on an imageshowing the basket capturing the object. The analyzed image can be asingular image captured by the imaging sensor or a stitched imagecreated by stitching together multiple images as introduced in the priorsection.

The image analysis performed by the object sizing system 800 includesperforming basket marker detection 1140 on the image. In this case, thebasketing apparatus is assumed to include visually distinctive markersoffset at fixed intervals from each other on the wires of the basket.The object sizing system 800 determines 1150 a distance from the objectto the imaging sensor of the endoscope capturing the image based on thelocations of the markers within the image.

In a variation on this technique, in addition to or in place of markerson the basketing apparatus, a guide wire may be inserted into thepatient and placed in front of the imaging sensor of the endoscope orbasketing apparatus, for example through the working channel of eitherof those instruments. The guide wire may similarly include visuallydistinctive markers. Based on known or tabulated information on theinsertion depth of the guide wire and/or based on any markers present onthe guide wire, the distance between the object and the imaging sensorof the endoscope can also be determined.

VIII. B. 2. H. Known Motion from a Robotically Controlled Basket

FIG. 11B shows an example flowchart illustrating a process of estimatingsize of an object using motion detection with a basket capturing theobject, according to one embodiment. As with the previous section, theobject sizing system 800 instructs a basket apparatus to capture 1181 anobject within a basket. In this embodiment, the basket apparatus is arobotically controlled basket receiving command instructions from theobject sizing system 800 for different movements (e.g., insertion,rotation, etc.) to capture the object. In one embodiment, afterreceiving 1182 notification that the object is captured within thebasket, the object sizing system 800 captures 1183 a sequence of imagesof the object and also receives robot data associated with the capturedimages. In the same or a different embodiment, the object sizing system800 captures 1183 a sequences of images of the object and also receivesrobot data associated with the captured images, regardless of whetherthe object is captured in the basket. Generally, however, these imageswill include at least some set of images containing at least a portionof the basket and at least a portion of the object. To acquire asequence of images of the object, the object sizing system 800 caninstruct the camera to move around the object to take images of theobject. The object sizing system 800 can also instruct the camera tostay fixed in position while taking images of the object, which can beeasier if the object is moving (e.g., moving according to changes of theendolumenal environment surrounding the object).

The object sizing system 800 performs 1184 image analysis to identifythe object. This may include segmenting the images to remove the basketfrom the images, leaving the object and any other data of interest.Alternatively, the images may be segmented to isolate the basket andobject so that they may be analyzed relative to each other. As thebasket may be moving between captured image frames, the robot and/or EMdata indicates real-time position and movement of the roboticallycontrolled basket, for example, distance of the basket from the camera,which can then be used to determine the size of the basket in the imageframes. This information can then be used to identify which portions ofthe image are due to the basket, which can then be segmented out fromthe images. The robot and/or EM data also indicates the real-time sizeof the basket, which can further provide additional useful information,such as an upper bound on the size of the object, which can, forexample, be a starting point in a more rigorous calculation of the sizeof the object.

The object sizing system 800 analyzes 1185 motion of the object andmotion of the imaging sensor as provided by the robot data throughoutthe sequence of the captured images. As described above, the robot datadescribes endoscope motion such as insertion, articulation as well asposition measurements of the object and the endoscope's imaging sensor.Thus, the robot data allows the object sizing system 800 to derive themotion of the imaging sensor. Once the motion of the imaging sensor isdetermined, this can subtracted out to isolate the motion of the objectby removing motion associated with the imaging sensor.

Once the motion of the object has been isolated, the object sizingsystem 800 can further analyze the motion of the object to identify itsstructure. As above, structure may include shape and estimated size. Forexample, if the object is rotating throughout the sequence of images,that rotation may provide a more complete 3D picture of the object'sshape that otherwise may not be able to be obtained due to limitationsin the repositionability of the imaging sensor. Specifically, the objectsizing system 800 determines 1186 an estimated size of the object. Asdescribed above in step 1884, the object sizing system 800 can usepixels of the remaining portions of the captured images after segmentingout the basket to determine an estimated size of the object.

VIII. B. 3 Illumination Pattern Analysis

FIG. 12 shows an example flowchart illustrating a process of estimatingsize of an object illuminated by structured light within a lumen of apatient's body, according to one embodiment. The object sizing system800 instructs the illumination source of the endoscope to illuminate1210 the object with structured light. The structured light may containa pattern that is projected on the object. The object sizing system 800is navigated 1120 so that the field of view of the endoscope's imagingsensor includes the object and the pattern of structured light projectedthereupon. The object sizing system 800 then instructs the imagingsensor to capture 1230 images of the object. The object sizing system800 analyzes 1240 how the reflection of the pattern is modified by beingprojected on and reflected by the object. Generally objects will havenon-planar structure along a plane perpendicular to the imaging sensorcentral axis, and so the reflection of the pattern will appeardifferently versus what is projected from the illumination source. Thereflection of the structured light may also vary depending on materialsof the object. As one example, objects that have different materialssuch as plastic or metal can have different reflection patterns based onsame structured light. As another example, objects that are composed ofdifferent proportions of the same materials (e.g., stones made up ofdifferent calcium concentrations) can also have different reflectionpatterns based on the same structured light. The object sizing system800 can analyze pattern features projected on the object based on asingle image. The object sizing system 800 can also analyze patternfeatures by combining multiple images captured from different views tocreate a complete 3D model based on robot data describing the differentviews. Changes in the pattern structure can be mathematically computedto determine the structure of the object. The object sizing system 800can then determine 1250 an estimated size or other aspects of thestructure of the object based on the analysis.

VIII. B. 4 Motion Detection with Robot Data or Optical Flow

FIG. 13 shows an example flowchart illustrating a process of estimatingsize of an object within a lumen of a patient's body employing motiondetection using robot data or optical flow techniques, according to oneembodiment. The object sizing system identifies 1310 an object within alumen of a patient's body using an endoscope. The object sizing system800 acquires 1320 a sequence of captured images of the object. Theobject sizing system 800 performs 1330 image analysis on the capturedimages by applying optical flow technique on object pixels. In moredetail, the object sizing system 800 isolates the identified objectshown in the captured images from the rest of the images, and appliesoptical flow technique only to the isolated object shown in the images.The pixels relating to identified object shown in the captured imagesmay have particular movement or position change that differdistinctively from those relating to the endolumenal environmentsurrounding the object. As one example, if the identified object is akidney stone positioned inside the kidney, the pixels of the stone shownin captured images may reflect movement of the stone according tomovement of the ureter. Performing optical flow on the specific pixelsdepicting the object can therefore provide beneficial information (fornavigation, analysis or otherwise) beyond that of optical flowperformed, for example, on images as a whole.

To isolate the object from the rest of a captured image, as one example,the object sizing system 800 may segment the object as described insubsection VIII. B. 2. H. The object sizing system 800 may also analyzethe captured image by detecting the rigid parts of the object.

While applying optical flow technique to perform image analysis, theobject sizing system 800 can use the identified object as a referencepoint combined with the description of the optical flow techniquesdescribed in patent application Ser. No. 15/268,238 that is herebyincorporated by reference in its entirety. For example, if optical flowof captured images is used to perform navigation of an instrument, theoptical flow analysis may be augmented by including optical flowanalysis of, specifically, the portions of the image depicting theobject to provide further information about the movement of theinstrument in the patient lumen.

Regardless of the approach used above, the object sizing system 800analyzes 1340 the motion of the object and imaging sensor relative tothe object throughout the sequence of the captured images. The objectsizing system 800 identifies 1350 a structure of the object based on theanalysis. The object sizing system 800 determines 1360 an estimated sizeof the object based on the identified structure.

IX. Machine Configuration for the Surgical Robotic System

More generally, the object sizing system techniques disclosed herein maybe implemented and performed in conjunction with an appropriatelyconfigured computer system. A processor within the computer system maycomprise one or more components to process electronic signals, and maycomprise one or more of a central processor unit, a video processor,logic circuitry, gate array logic, filed programmable gate array,integrated circuit, or application specific integrated circuit. Thecomputer system includes a central processing unit (CPU, also“processor” and “computer processor” herein), which can be a single coreor multi core processor, or a plurality of processors for parallelprocessing. The CPU can execute a sequence of machine-readableinstructions, which can be embodied in a program or software. Theinstructions may be stored in a memory location. Examples of operationsperformed by the CPU can include fetch, decode, execute, and writeback.The CPU can be part of a circuit, such as an integrated circuit. One ormore other components of the system can be included in the circuit. Insome cases, the circuit comprises an application specific integratedcircuit (ASIC).

The computer system may also include one or more of memory or memorylocations (e.g., random-access memory, read-only memory, flash memory),electronic storage units (e.g., hard disk), communication interfaces(e.g., network adapters) for communicating with one or more othersystems, and peripheral devices, such as caches, other memory, datastorage and/or electronic display adapters. The memory, storage unit,interface, and peripheral devices are in communication with the CPUthrough a communication bus, such as a motherboard.

The storage unit can be a data storage unit (or data repository) forstoring data. The computer system can be operatively coupled to acomputer network (“network”) with the aid of the communicationinterface. The network can be the Internet, an internet and/or extranet,or an intranet and/or extranet that is in communication with theInternet. The network in some cases is a telecommunication and/or datanetwork, and can include can include one or more computer servers. Thestorage unit can store files, such as drivers, libraries and savedprograms. The storage unit can store user data, e.g., user preferencesand user programs. The computer system in some cases can include one ormore additional data storage units that are external to the computersystem, such as located on a remote server that is in communication withthe computer system through an intranet or the Internet.

Methods as described herein can be implemented by way of machine (e.g.,computer processor) executable code stored on an electronic storagelocation of the computer system, such as, for example, on the memory orelectronic storage unit. The machine executable or machine readable codecan be provided in the form of software. During use, the code can beexecuted by the processor. In some cases, the code can be retrieved fromthe storage unit and stored on the memory for ready access by theprocessor. In some situations, the electronic storage unit can beprecluded, and machine-executable instructions are stored on memory.

The code can be pre-compiled and configured for use with a machine havea processer adapted to execute the code, or can be compiled duringruntime. The code can be supplied in a programming language that can beselected to enable the code to execute in a pre-compiled or as-compiledfashion.

Methods and systems of the present disclosure can be implemented by wayof one or more methods. A method can be implemented with a processor asdescribed herein, for example by way of software upon execution by oneor more computer processors.

X. Additional Considerations

Upon reading this disclosure, those of skill in the art will appreciatestill additional alternative structural and functional designs throughthe disclosed principles herein. Thus, while particular embodiments andapplications have been illustrated and described, it is to be understoodthat the disclosed embodiments are not limited to the preciseconstruction and components disclosed herein. Various modifications,changes and variations, which will be apparent to those skilled in theart, may be made in the arrangement, operation and details of the methodand apparatus disclosed herein without departing from the spirit andscope defined in the appended claims.

As used herein any reference to “one embodiment” or “an embodiment”means that a particular element, feature, structure, or characteristicdescribed in connection with the embodiment is included in at least oneembodiment. The appearances of the phrase “in one embodiment” in variousplaces in the specification are not necessarily all referring to thesame embodiment.

Some embodiments may be described using the expression “coupled” and“connected” along with their derivatives. For example, some embodimentsmay be described using the term “coupled” to indicate that two or moreelements are in direct physical or electrical contact. The term“coupled,” however, may also mean that two or more elements are not indirect contact with each other, but yet still co-operate or interactwith each other. The embodiments are not limited in this context unlessotherwise explicitly stated.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,method, article, or apparatus that comprises a list of elements is notnecessarily limited to only those elements but may include otherelements not expressly listed or inherent to such process, method,article, or apparatus. Further, unless expressly stated to the contrary,“or” refers to an inclusive or and not to an exclusive or. For example,a condition A or B is satisfied by any one of the following: A is true(or present) and B is false (or not present), A is false (or notpresent) and B is true (or present), and both A and B are true (orpresent). In addition, use of the “a” or “an” are employed to describeelements and components of the embodiments herein. This is done merelyfor convenience and to give a general sense of the invention. Thisdescription should be read to include one or at least one and thesingular also includes the plural unless it is obvious that it is meantotherwise.

What is claimed is:
 1. A method for sizing an object within a patient,the method comprising: navigating an elongate body to a location withinthe patient and proximate to the object, the elongate body comprising animaging sensor for capturing images of the object and a basket forcapturing the object; capturing sequential images of the object and thebasket with the imaging sensor while navigating the elongate body intoat least one of a different orientation and a different position;accessing, with a processor, a set of robot data indicative of theorientation and position of the elongate body during the capture of thesequential images, wherein the accessed set of robot data comprises datafrom which a position and size of the basket can be determined; andanalyzing, with the processor, the sequential images to estimate a sizeof the object based on: identifying those images depicting at least aportion of the object and at least a portion of the basket; anddetermining the estimated size of the object based on the position andthe size of the basket relative to the object in the identified images.2. The method of claim 1, wherein analyzing the sequential imagescomprises segmenting the sequential images to remove the basket from theanalyzed images.
 3. The method of claim 2, wherein analyzing thesequential images comprises determining a structure of the object basedon the segmented images.
 4. The method of claim 1, further comprisingcapturing the object with the basket.
 5. The method of claim 4, whereincapturing the sequential images occurs before capturing the object withthe basket.
 6. The method of claim 4, wherein capturing the sequentialimages occurs after capturing the object with the basket.
 7. The methodof claim 4, wherein analyzing the sequential images comprises: detectinga basket marker in the sequential images; and determining a distance tothe object based on the detected basket marker, wherein determining theestimated size of the object is further based on the distance to theobject.
 8. The method of claim 1, wherein the set of robot datacomprises at least one of data collected from a robotic manipulator anddata collected from an electromagnetic sensor coupled to the elongatebody.
 9. The method of claim 1, further comprising inserting a guidewire into the patient to measure a distance from the imaging sensor tothe object.
 10. The method of claim 1, wherein analyzing the sequentialimages comprises: combining the sequential images via image stitchingtechniques; identifying a structure of the object; determining anestimated size of the identified structure of the object; anddetermining the estimated size of the object further based on theestimated size of the structure of the object.
 11. The method of claim10, wherein the structure of the object comprises at least one of anouter axial diameter and a radius of the object.
 12. A system for sizingan object within a patient, the system comprising: an elongate bodycomprising an imaging sensor for capturing images of the object and abasket for capturing the object; and a computer comprising instructionsthat, when executed by a processor of the computer, cause the computerto: navigate the elongate body into the patient and to a locationproximate to the object, the navigating enabled by a roboticmanipulator; capture sequential images of the object and the basket withthe imaging sensor while navigating the elongate body into at least oneof a different orientation and a different position; access a set ofrobotic data describing the changing orientation and position of theelongate body during the capture of the sequential images; and analyzethe sequential images to estimate a size of the object based on:identifying those images depicting at least a portion of the object andat least a portion of the basket; and determining the estimated size ofthe object based on the position and the size of the basket relative tothe object in the identified images.
 13. The system of claim 12, whereinthe instructions, when executed by the processor, cause the computer toanalyze the sequential images based on segmenting the sequential imagesto remove the basket from the analyzed images.
 14. The system of claim12, wherein the instructions, when executed by the processor, cause thecomputer to instruct the system to capture the object with the basket.15. The system of claim 14, wherein the capturing of the sequentialimages occurs before the capturing of the object with the basket. 16.The system of claim 14, wherein the capturing of the sequential imagesoccurs after the capturing of the object with the basket.
 17. The systemof claim 14, wherein instructions, when executed by theprocessor, causethe computer to analyze the sequential images based on: detecting abasket marker in the sequential images; and determining a distance tothe object based on the detected basket marker, wherein determining theestimated size of the object is further based on the distance to theobject.
 18. The system of claim 12, wherein the set of robot datacomprises at least one of data collected from a robotic manipulator anddata collected from an electromagnetic sensor coupled to the elongatebody.
 19. The system of claim 12, further comprising: a guide wire,wherein the instructions, when executed by the processor, cause thecomputer to: insert the guide wire into the patient to measure adistance from the imaging sensor to the object.
 20. The system of claim12, wherein the instructions, when executed by theprocessor, cause thecomputer to analyze the sequential images based on: combining thesequential images via image stitching techniques; identifying astructure of the object; and determining the estimated size of theobject further based on the identified structure.